The invention is directed to an improved approach for testing and verifying user interfaces.
Most computing devices, applications, and complex tools rely upon a user interface to interact with, receive input from, and provide information to users. There are many types of user interfaces. Common approaches to implement user interfaces include the graphical user interface (GUI), character user interface (CUI), and web-based user interfaces.
Like any other development process for a complex design, it is important to ensure that the process for developing a user interface involves adequate testing and verification of the performance and functionality of the interface components. In the field of computer science, GUI software testing is the process of testing a product that uses a Graphical User Interface (GUI), to make sure it meets its written specifications. This is normally done through the use of a variety of test cases, in addition to ad-hoc methods involving human interaction.
To generate a comprehensive set of test cases however, the test designer must be certain that their suite covers all the functionality of the system and also has to be sure that the suite fully exercises the GUI itself. The difficulty in accomplishing this task is twofold: one has to deal with domain size and then one has to deal with sequences. In addition, the tester faces more difficulty when they have to do regression testing.
The size problem can be easily illustrated. Unlike a CLI (Command Line Interface) system, a GUI has many operations that need to be tested. A very small program such as Microsoft WordPad has at least 325 possible GUI operations. In a large program, the number of operations can easily be an order of magnitude larger.
One possible testing approach is to perform “capture/playback.” Capture/playback is an approach where the system screen is “captured” as a bitmapped graphic at various times during system testing. This capturing allowed the tester to “playback” the testing process and compares the screens at the output phase of the test with expected screens. This validation could be automated since the screens would be identical if the case passed and different if the case failed.
However, there are significant problems when one tries to implement capture/playback in a complex system, such as most GUI based systems. The most obvious problem one finds is that the screen in a GUI system may look different while the state of the underlying system is the same, making automated validation extremely difficult. This is because a GUI allows graphical objects to vary in appearance and placement on the screen. For example, fonts may be different and window colors or sizes may vary, even though the system output is basically the same. This would be obvious to a user, but not obvious to an automated validation system.
To combat this and other problems, another approach is to go ‘under the hood’ and collected GUI interaction data from the underlying windowing system. By capturing the window ‘events’ into logs, the interactions with the system are now in a format that is decoupled from the appearance of the GUI. The advantage is that the event streams are captured. Unfortunately, significant filtering of the event streams becomes necessary when using this approach since the streams of events are usually very detailed and most events are not directly relevant to the functionality being targeted by the GUI test.
In practice, it often also turns out that the overhead of maintaining regression tests across multiple software releases is sufficient to discourage the use of automated GUI testing in the first place, in spite of its otherwise apparent advantages and necessity. This essentially means that existing GUI tests are not easily reusable, and most be extensively modified before being used for varied GUI configurations.
Another possible approach to running tests on a GUI is to build a driver into the GUI so that commands or events can be sent to the software from another program. This method of directly sending events to and receiving events from a system is highly desirable when testing, since the input and output testing can be fully automated and user error is eliminated. However, this method is significantly limited by the fact that software GUIs tend to change from software release to release. Even though the underlying software major functionality may not change, the fact that the GUI itself changes is often sufficient to induce a major maintenance overhead requiring manual regeneration of tests. Further issues arise in that particular solutions are limited to a single specific application only, such that reuse of the test infrastructure across multiple similar or differing applications is not possible.
Therefore, it is clear that there is a need for an improved approach to perform testing of user interfaces. A system and method is described in which the state of the art in automated software applications is significantly improved. According to some embodiments, interface testing is implemented and based upon a verification language and a verification environment.
Other and additional objects, features, and advantages of the invention are described in the detailed description, figures, and claims.